SherbyNet - /page/wikibook/entry.mu

Micron Document

<!DOCTYPE html>

<title>Mutation testing</title>

</head>

Mutation testing

</h1>

</header>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hatnote{font-style:italic}.mw-parser-output div.hatnote{padding-left:1.6em;margin-bottom:0.5em}.mw-parser-output .hatnote i{font-style:normal}.mw-parser-output .hatnote+link+.hatnote{margin-top:-0.5em}@media print{body.ns-0 .mw-parser-output .hatnote{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div role="note" class="hatnote navigation-not-searchable">"Mutation analysis" redirects here. For the biological term, see <a href="Gene_mutation_analysis" class="mw-redirect" title="Gene mutation analysis">Gene mutation analysis</a>.</div>

Mutation testing (or mutation analysis or program mutation) is used to design new software tests and evaluate the quality of existing software tests. Mutation testing involves modifying a program in small ways.<a href="#cite_note-DLS1978-1">[1]</a> Each mutated version is called a mutant and tests detect and reject mutants by causing the behaviour of the original version to differ from the mutant. This is called killing the mutant. Test suites are measured by the percentage of mutants that they kill. New tests can be designed to kill additional mutants. Mutants are based on well-defined mutation operators that either mimic typical programming errors (such as using the wrong operator or variable name) or force the creation of valuable tests (such as dividing each expression by zero). The purpose is to help the tester develop effective tests or locate weaknesses in the test data used for the program or in sections of the code that are seldom or never accessed during <a href="Execution_(computers)" class="mw-redirect" title="Execution (computers)">execution</a>. Mutation testing is a form of <a href="White-box_testing" title="White-box testing">white-box testing</a>.<a href="#cite_note-2">[2]</a><a href="#cite_note-3">[3]</a>

<div class="mw-heading mw-heading2"><h2 id="Introduction">Introduction</h2></div>

Most of this article is about "program mutation", in which the program is modified. A more general definition of mutation analysis is using well-defined rules defined on syntactic structures to make systematic changes to software artifacts.<a href="#cite_note-AmmannOffutt2008-4">[4]</a> Mutation analysis has been applied to other problems, but is usually applied to testing. So mutation testing is defined as using mutation analysis to design new software tests or to evaluate existing software tests.<a href="#cite_note-AmmannOffutt2008-4">[4]</a> Thus, mutation analysis and testing can be applied to design models, specifications, databases, tests, XML, and other types of software artifacts, although program mutation is the most common.<a href="#cite_note-5">[5]</a>

<div class="mw-heading mw-heading2"><h2 id="Overview">Overview</h2></div>

Tests can be created to verify the correctness of the implementation of a given software system, but the creation of tests still poses the question whether the tests are correct and sufficiently cover the requirements that have originated the implementation.<a href="#cite_note-6">[6]</a> (This technological problem is itself an instance of a deeper philosophical problem named "<a href="Quis_custodiet_ipsos_custodes%3F" title="Quis custodiet ipsos custodes?">Quis custodiet ipsos custodes?</a>" ["Who will guard the guards?"].) The idea behind mutation testing is that if a mutant is introduced, this normally causes a bug in the program's functionality which the tests should find. This way, the tests are tested. If a mutant is not detected by the test suite, this typically indicates that the test suite is unable to locate the faults represented by the mutant, but it can also indicate that the mutation introduces no faults, that is, the mutation is a valid change that does not affect functionality. One (common) way a mutant can be valid is that the code that has been changed is "dead code" that is never executed.

For mutation testing to function at scale, a large number of mutants are usually introduced, leading to the compilation and execution of an extremely large number of copies of the program. This problem of the expense of mutation testing had reduced its practical use as a method of software testing. However, the increased use of <a href="Object_oriented_programming_language" class="mw-redirect" title="Object oriented programming language">object oriented programming languages</a> and <a href="Unit_testing" title="Unit testing">unit testing</a> frameworks has led to the creation of mutation testing tools that test individual portions of an application.

<div class="mw-heading mw-heading2"><h2 id="Goals">Goals</h2></div>

The goals of mutation testing are multiple:

<ul><li>identify weakly tested pieces of code (those for which mutants are not killed)<a href="#cite_note-DLS1978-1">[1]</a></li>

<li>identify weak tests (those that never kill mutants)<a href="#cite_note-Smith2008-7">[7]</a></li>

<li>compute the mutation score,<a href="#cite_note-AmmannOffutt2008-4">[4]</a> the mutation score is the number of mutants killed / total number of mutants.</li>

<li>learn about error propagation and state infection in the program</li></ul>

<div class="mw-heading mw-heading2"><h2 id="History">History</h2></div>

Mutation testing was originally proposed by Richard Lipton as a student in 1971,<a href="#cite_note-mutation2000-8">[8]</a> and first developed and published by DeMillo, Lipton and Sayward.<a href="#cite_note-DLS1978-1">[1]</a> The first implementation of a mutation testing tool was by Timothy Budd as part of his <a href="PhD" class="mw-redirect" title="PhD">PhD</a> work (titled Mutation Analysis) in 1980 from <a href="Yale_University" title="Yale University">Yale University</a>.<a href="#cite_note-Budd1980-9">[9]</a>

Recently, with the availability of massive computing power, there has been a resurgence of mutation analysis within the computer science community, and work has been done to define methods of applying mutation testing to <a href="Object_oriented_programming_language" class="mw-redirect" title="Object oriented programming language">object oriented programming languages</a> and non-procedural languages such as <a href="XML" title="XML">XML</a>, <a href="Symbolic_Model_Verification" class="mw-redirect" title="Symbolic Model Verification">SMV</a>, and <a href="Finite-state_machine" title="Finite-state machine">finite-state machines</a>.

In 2004, a company called Certess Inc. (now part of <a href="Synopsys" title="Synopsys">Synopsys</a>) extended many of the principles into the hardware verification domain. Whereas mutation analysis only expects to detect a difference in the output produced, Certess extends this by verifying that a checker in the testbench will actually detect the difference. This extension means that all three stages of verification, namely: activation, propagation, and detection are evaluated. They called this functional qualification.

<a href="Fuzzing" title="Fuzzing">Fuzzing</a> can be considered to be a special case of mutation testing. In fuzzing, the messages or data exchanged inside communication interfaces (both inside and between software instances) are mutated to catch failures or differences in processing the data. <a href="Codenomicon" class="mw-redirect" title="Codenomicon">Codenomicon</a><a href="#cite_note-10">[10]</a> (2001) and <a href="Mu_Dynamics" class="mw-redirect" title="Mu Dynamics">Mu Dynamics</a> (2005) evolved fuzzing concepts to a fully stateful mutation testing platform, complete with monitors for thoroughly exercising protocol implementations.

<div class="mw-heading mw-heading2"><h2 id="Mutation_testing_overview">Mutation testing overview</h2></div>

Mutation testing is based on two hypotheses. The first is the competent programmer hypothesis. This hypothesis states that competent programmers write programs that are close to being correct.<a href="#cite_note-DLS1978-1">[1]</a> "Close" is intended to be based on behavior, not syntax. The second hypothesis is called the coupling effect. The coupling effect asserts that simple faults can cascade or couple to form other emergent faults.<a href="#cite_note-Offut1992-11">[11]</a><a href="#cite_note-ABDLS1979-12">[12]</a>

Subtle and important faults are also revealed by higher-order mutants, which further support the coupling effect.<a href="#cite_note-JH2008-13">[13]</a><a href="#cite_note-Umar2008-14">[14]</a><a href="#cite_note-Smith2008-7">[7]</a><a href="#cite_note-PP2009-15">[15]</a><a href="#cite_note-Anderson2011-16">[16]</a> Higher-order mutants are enabled by creating mutants with more than one mutation.

Mutation testing is done by selecting a set of mutation operators and then applying them to the source program one at a time for each applicable piece of the source code. The result of applying one mutation operator to the program is called a mutant. If the test suite is able to detect the change (i.e. one of the tests fails), then the mutant is said to be killed.

For example, consider the following C++ code fragment:

<div class="mw-highlight mw-highlight-lang-cpp mw-content-ltr" dir="ltr"><pre>if (a && b) {

c = 1;

} else {

c = 0;

}

</pre></div>

The condition mutation operator would replace <code>&&</code> with <code>||</code> and produce the following mutant:

<div class="mw-highlight mw-highlight-lang-cpp mw-content-ltr" dir="ltr"><pre>if (a || b) {

c = 1;

} else {

c = 0;

}

</pre></div>

Now, for the test to kill this mutant, the following three conditions should be met:

<ol><li>A test must reach the mutated statement.</li>

<li>Test input data should infect the program state by causing different program states for the mutant and the original program. For example, a test with <code>a = 1</code> and <code>b = 0</code> would do this.</li>

<li>The incorrect program state (the value of 'c') must propagate to the program's output and be checked by the test.</li></ol>

These conditions are collectively called the RIP model.<a href="#cite_note-mutation2000-8">[8]</a>

Weak mutation testing (or weak mutation coverage) requires that only the first and second conditions are satisfied. Strong mutation testing requires that all three conditions are satisfied. Strong mutation is more powerful, since it ensures that the test suite can really catch the problems. Weak mutation is closely related to <a href="Code_coverage" title="Code coverage">code coverage</a> methods. It requires much less computing power to ensure that the test suite satisfies weak mutation testing than strong mutation testing.

However, there are cases where it is not possible to find a test case that could kill this mutant. The resulting program is behaviorally equivalent to the original one. Such mutants are called equivalent mutants.

Equivalent mutants detection is one of biggest obstacles for practical usage of mutation testing. The effort needed to check if mutants are equivalent or not can be very high even for small programs.<a href="#cite_note-17">[17]</a> A 2014 systematic literature review of a wide range of approaches to overcome the Equivalent Mutant Problem<a href="#cite_note-18">[18]</a> identified 17 relevant techniques (in 22 articles) and three categories of techniques: detecting (DEM); suggesting (SEM); and avoiding equivalent mutant generation (AEMG). The experiment indicated that Higher Order Mutation in general and JudyDiffOp strategy in particular provide a promising approach to the Equivalent Mutant Problem.

In addition to equivalent mutants, there are subsumed mutants which are mutants that exist in the same source code location as another mutant, and are said to be "subsumed" by the other mutant. Subsumed mutants are not visible to a mutation testing tool, and do not contribute to coverage metrics. For example, let's say you have two mutants, A and B, that both change a line of code in the same way. Mutant A is tested first, and the result is that the code is not working correctly. Mutant B is then tested, and the result is the same as with mutant A. In this case, Mutant B is considered to be subsumed by Mutant A, since the result of testing Mutant B is the same as the result of testing Mutant A. Therefore, Mutant B does not need to be tested, as the result will be the same as Mutant A.

<div class="mw-heading mw-heading2"><h2 id="Mutation_operators">Mutation operators</h2></div>

To make syntactic changes to a program, a mutation operator serves as a guideline that substitutes portions of the source code. Given that mutations depend on these operators, scholars have created a collection of mutation operators to accommodate different programming languages, like Java. The effectiveness of these mutation operators plays a pivotal role in mutation testing.<a href="#cite_note-:1-19">[19]</a>

Many mutation operators have been explored by researchers. Here are some examples of mutation operators for imperative languages:

<ul><li>Statement deletion</li>

<li>Statement duplication or insertion, e.g. <code>goto fail;</code><a href="#cite_note-20">[20]</a></li>

<li>Replacement of Boolean subexpressions with true and false</li>

<li>Replacement of some arithmetic operations with others, e.g. <code>+</code> with <code>*</code>, <code>-</code> with <code>/</code></li>

<li>Replacement of some Boolean relations with others, e.g. <code>></code> with <code>>=</code>, <code>==</code> and <code><=</code></li>

<li>Replacement of variables with others from the same scope (variable types must be compatible)</li>

<li>Remove method body.<a href="#cite_note-21">[21]</a></li></ul>

These mutation operators are also called traditional mutation operators.

There are also mutation operators for object-oriented languages,<a href="#cite_note-22">[22]</a> for concurrent constructions,<a href="#cite_note-23">[23]</a> complex objects like containers,<a href="#cite_note-24">[24]</a> etc.

<div class="mw-heading mw-heading3"><h3 id="Types_of_mutation_operators">Types of mutation operators</h3></div>

Operators for containers are called class-level mutation operators. Operators at the class level alter the program's structure by adding, removing, or changing the expressions being examined. Specific operators have been established for each category of changes.<a href="#cite_note-:1-19">[19]</a> For example, the muJava tool offers various class-level mutation operators such as Access Modifier Change, Type Cast Operator Insertion, and Type Cast Operator Deletion. Mutation operators have also been developed to perform security vulnerability testing of programs.<a href="#cite_note-25">[25]</a>

Apart from the class-level operators, MuJava also includes method-level mutation operators, referred to as traditional operators. These traditional operators are designed based on features commonly found in procedural languages. They carry out changes to statements by adding, substituting, or removing primitive operators. These operators fall into six categories: Arithmetic operators, Relational operators, Conditional operators, Shift operators, Logical operators and Assignment operators.<a href="#cite_note-:1-19">[19]</a>

<div class="mw-heading mw-heading2"><h2 id="Types_of_mutation_testing">Types of mutation testing</h2></div>

There are three types of mutation testing;

<div class="mw-heading mw-heading3"><h3 id="Statement_mutation">Statement mutation</h3></div>

Statement mutation is a process where a block of code is intentionally modified by either deleting or copying certain statements. Moreover, it allows for the reordering of statements within the code block to generate various sequences.<a href="#cite_note-:0-26">[26]</a> This technique is crucial in software testing as it helps identify potential weaknesses or errors in the code. By deliberately making changes to the code and observing how it behaves, developers can uncover hidden bugs or flaws that might go unnoticed during regular testing.<a href="#cite_note-27">[27]</a> Statement mutation is like a diagnostic tool that provides insights into the code's robustness and resilience, helping programmers improve the overall quality and reliability of their software.

For example, in the code snippet below, entire 'else' section is removed:

<div class="mw-highlight mw-highlight-lang-cpp mw-content-ltr" dir="ltr"><pre>function checkCredentials(username, password) {

if (username === "admin" && password === "password") {

return true;

}

}

</pre></div>

<div class="mw-heading mw-heading3"><h3 id="Value_mutation">Value mutation</h3></div>

Value mutation occurs when modification is executed to the parameter and/or constant values within the code. This typically involves adjusting the values by adding or subtracting 1, but it can also involve making more substantial changes to the values. The specific alterations made during value mutation include two main scenarios:

Firstly, there's the transformation from a small value to a higher value. This entails replacing a small value in the code with a larger one. The purpose of this change is to assess how the code responds when it encounters larger inputs. It helps ensure that the code can accurately and efficiently process these larger values without encountering errors or unexpected issues.<a href="#cite_note-:0-26">[26]</a>

Conversely, the second scenario involves changing a higher value to a smaller one. In this case, we replace a higher value within the code with a smaller value. This test aims to evaluate how the code handles smaller inputs. Ensuring that the code performs correctly with smaller values is essential to prevent unforeseen problems or errors when dealing with such input data.<a href="#cite_note-:0-26">[26]</a>

For example:

<div class="mw-highlight mw-highlight-lang-cpp mw-content-ltr" dir="ltr"><pre>// Original code

function multiplyByTwo(value) {

return value * 2;

}

// Value mutation: Small value to higher value

function multiplyByTwoMutation1(value) {

}

// Value mutation: Higher value to small value

function multiplyByTwoMutation2(value) {

return value / 10;

}

</pre></div>

<div class="mw-heading mw-heading3"><h3 id="Decision_mutation">Decision mutation</h3></div>

Decision mutation testing centers on the identification of design errors within the code, with a particular emphasis on detecting flaws or weaknesses in the program's decision-making logic. This method involves deliberately altering arithmetic and logical operators to expose potential issues.<a href="#cite_note-:0-26">[26]</a> By manipulating these operators, developers can systematically evaluate how the code responds to different decision scenarios. This process helps ensure that the program's decision-making pathways are robust and accurate, preventing costly errors that could arise from faulty logic. Decision mutation testing serves as a valuable tool in software development, enabling developers to enhance the reliability and effectiveness of their decision-making code segments.

For example:

<div class="mw-highlight mw-highlight-lang-cpp mw-content-ltr" dir="ltr"><pre>// Original code

function isPositive(number) {

return number > 0;

}

// Decision mutation: Changing the comparison operator

function isPositiveMutation1(number) {

return number >= 0;

}

// Decision mutation: Negating the result

function isPositiveMutation2(number) {

return !(number > 0);

}

</pre></div>

<ul><li><a href="Bebugging" title="Bebugging">Bebugging</a> (or fault seeding)</li>

<li><a href="Sanity_testing" class="mw-redirect" title="Sanity testing">Sanity testing</a></li>

<li><a href="Fault_injection" title="Fault injection">Fault injection</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist">

<li id="cite_note-DLS1978-1">^ <a href="#cite_ref-DLS1978_1-0">a</a> <a href="#cite_ref-DLS1978_1-1">b</a> <a href="#cite_ref-DLS1978_1-2">c</a> <a href="#cite_ref-DLS1978_1-3">d</a> Richard A. DeMillo, Richard J. Lipton, and Fred G. Sayward. Hints on test data selection: Help for the practicing programmer. IEEE Computer, 11(4):34-41. April 1978.

</li>

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite id="CITEREFOstrand2002" class="citation cs2">Ostrand, Thomas (2002), <a rel="nofollow" class="external text" href="https://onlinelibrary.wiley.com/doi/abs/10.1002/0471028959.sof378">"White-Box Testing"</a>, Encyclopedia of Software Engineering, American Cancer Society, <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1002%2F0471028959.sof378">10.1002/0471028959.sof378</a>, <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-471-02895-6</bdi>, retrieved 2021-03-16</cite>

</li>

<li id="cite_note-3"><a href="#cite_ref-3">^</a> <cite id="CITEREFMisra2003" class="citation book cs1">Misra, S. (2003). <a rel="nofollow" class="external text" href="https://ieeexplore.ieee.org/document/1226246">"Evaluating four white-box test coverage methodologies"</a>. CCECE 2003 - Canadian Conference on Electrical and Computer Engineering. Toward a Caring and Humane Technology (Cat. No.03CH37436). Vol. 3. Montreal, Que., Canada: IEEE. pp. 1739–1742. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FCCECE.2003.1226246">10.1109/CCECE.2003.1226246</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-7803-7781-3</bdi>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:62549502">62549502</a>.</cite>

</li>

<li id="cite_note-AmmannOffutt2008-4">^ <a href="#cite_ref-AmmannOffutt2008_4-0">a</a> <a href="#cite_ref-AmmannOffutt2008_4-1">b</a> <a href="#cite_ref-AmmannOffutt2008_4-2">c</a> Paul Ammann and Jeff Offutt. Introduction to Software Testing. Cambridge University Press, 2008.

</li>

<li id="cite_note-5"><a href="#cite_ref-5">^</a> <cite id="CITEREFJiaHarman2009" class="citation journal cs1">Jia, Yue; <a href="Mark_Harman_(computer_scientist)" title="Mark Harman (computer scientist)">Harman, Mark</a> (September 2009). <a rel="nofollow" class="external text" href="https://web.archive.org/web/20171204061252/https://pdfs.semanticscholar.org/3277/8a2eb4c74cd437e922ac1eb6a1477dfcb925.pdf">"An Analysis and Survey of the Development of Mutation Testing"</a> (PDF). IEEE Transactions on Software Engineering. 37 (5): 649–678. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTSE.2010.62">10.1109/TSE.2010.62</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:6853229">6853229</a>. Archived from <a rel="nofollow" class="external text" href="https://pdfs.semanticscholar.org/3277/8a2eb4c74cd437e922ac1eb6a1477dfcb925.pdf">the original</a> (PDF) on 2017-12-04.</cite>

</li>

<li id="cite_note-6"><a href="#cite_ref-6">^</a> <cite id="CITEREFDassoFunes2007" class="citation book cs1">Dasso, Aristides; Funes, Ana (2007). Verification, Validation and Testing in Software Engineering. Idea Group Inc. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-1591408512</bdi>.</cite>

</li>

<li id="cite_note-Smith2008-7">^ <a href="#cite_ref-Smith2008_7-0">a</a> <a href="#cite_ref-Smith2008_7-1">b</a> Smith B., "On Guiding Augmentation of an Automated Test Suite via Mutation Analysis," 2008

</li>

<li id="cite_note-mutation2000-8">^ <a href="#cite_ref-mutation2000_8-0">a</a> <a href="#cite_ref-mutation2000_8-1">b</a> <a rel="nofollow" class="external text" href="http://cs.gmu.edu/~offutt/rsrch/papers/mut00.pdf">Mutation 2000: Uniting the Orthogonal</a> <a rel="nofollow" class="external text" href="https://web.archive.org/web/20110928184359/http://cs.gmu.edu/~offutt/rsrch/papers/mut00.pdf">Archived</a> 2011-09-28 at the <a href="Wayback_Machine" title="Wayback Machine">Wayback Machine</a> by <a href="Jeff_Offutt" title="Jeff Offutt">A. Jefferson Offutt</a> and Roland H. Untch.

</li>

<li id="cite_note-Budd1980-9"><a href="#cite_ref-Budd1980_9-0">^</a> Tim A. Budd, Mutation Analysis of Program Test Data. PhD thesis, Yale University New Haven CT, 1980.

</li>

<li id="cite_note-10"><a href="#cite_ref-10">^</a> <a rel="nofollow" class="external text" href="http://www.codenomicon.com/resources/publications.shtml">Kaksonen, Rauli. A Functional Method for Assessing Protocol Implementation Security (Licentiate thesis). Espoo. 2001.</a>

</li>

<li id="cite_note-Offut1992-11"><a href="#cite_ref-Offut1992_11-0">^</a> A. Jefferson Offutt. 1992. Investigations of the software testing coupling effect. ACM Trans. Softw. Eng. Methodol. 1, 1 (January 1992), 5-20.

</li>

<li id="cite_note-ABDLS1979-12"><a href="#cite_ref-ABDLS1979_12-0">^</a> A. T. Acree, T. A. Budd, R. A. DeMillo, R. J. Lipton, and F. G. Sayward, "Mutation Analysis," Georgia Institute of Technology, Atlanta, Georgia, Technique Report GIT-ICS-79/08, 1979.

</li>

<li id="cite_note-JH2008-13"><a href="#cite_ref-JH2008_13-0">^</a> Yue Jia; Harman, M., "Constructing Subtle Faults Using Higher Order Mutation Testing," Source Code Analysis and Manipulation, 2008 Eighth IEEE International Working Conference on , vol., no., pp.249,258, 28-29 Sept. 2008

</li>

<li id="cite_note-Umar2008-14"><a href="#cite_ref-Umar2008_14-0">^</a> Maryam Umar, "An Evaluation of Mutation Operators For Equivalent Mutants," MS Thesis, 2006

</li>

<li id="cite_note-PP2009-15"><a href="#cite_ref-PP2009_15-0">^</a> Polo M. and Piattini M., "Mutation Testing: practical aspects and cost analysis," University of Castilla-La Mancha (Spain), Presentation, 2009

</li>

<li id="cite_note-Anderson2011-16"><a href="#cite_ref-Anderson2011_16-0">^</a> Anderson S., "Mutation Testing", the University of Edinburgh, School of Informatics, Presentation, 2011

</li>

<li id="cite_note-17"><a href="#cite_ref-17">^</a> P. G. Frankl, S. N. Weiss, and C. Hu. All-uses versus mutation testing: An experimental comparison of effectiveness. Journal of Systems and Software, 38:235–253, 1997.

</li>

<li id="cite_note-18"><a href="#cite_ref-18">^</a> <a rel="nofollow" class="external text" href="http://madeyski.e-informatyka.pl/download/Madeyski13TSE.pdf">Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation</a> by L. Madeyski, W. Orzeszyna, R. Torkar, M. Józala. IEEE Transactions on Software Engineering

</li>

<li id="cite_note-:1-19">^ <a href="#cite_ref-:1_19-0">a</a> <a href="#cite_ref-:1_19-1">b</a> <a href="#cite_ref-:1_19-2">c</a> <cite id="CITEREFHamimouneFalah2016" class="citation book cs1">Hamimoune, Soukaina; Falah, Bouchaib (24 September 2016). "Mutation testing techniques: A comparative study". <a rel="nofollow" class="external text" href="https://web.archive.org/web/20180619222840/https://ieeexplore.ieee.org/document/7745368/">2016 International Conference on Engineering & MIS (ICEMIS)</a>. pp. 1–9. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FICEMIS.2016.7745368">10.1109/ICEMIS.2016.7745368</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-1-5090-5579-1</bdi>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:24301702">24301702</a>. Archived from the original on 19 June 2018. Retrieved 2023-10-08.</cite><code class="cs1-code">{{cite book}}</code>: CS1 maint: bot: original URL status unknown (link)

</li>

<li id="cite_note-20"><a href="#cite_ref-20">^</a> <a rel="nofollow" class="external text" href="https://www.imperialviolet.org/2014/02/22/applebug.html">Apple's SSL/TLS bug</a> by Adam Langley.

</li>

<li id="cite_note-21"><a href="#cite_ref-21">^</a> <cite id="CITEREFNiedermayrJuergensWagner2016" class="citation book cs1">Niedermayr, Rainer; Juergens, Elmar; Wagner, Stefan (2016-05-14). <a rel="nofollow" class="external text" href="https://doi.org/10.1145/2896941.2896944">"Will my tests tell me if I break this code?"</a>. Proceedings of the International Workshop on Continuous Software Evolution and Delivery. CSED '16. Austin, Texas: Association for Computing Machinery. pp. 23–29. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1611.07163">1611.07163</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1145%2F2896941.2896944">10.1145/2896941.2896944</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-1-4503-4157-8</bdi>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:9213147">9213147</a>.</cite>

</li>

<li id="cite_note-22"><a href="#cite_ref-22">^</a> <a rel="nofollow" class="external text" href="http://www.cs.gmu.edu/~offutt/rsrch/papers/mujava.pdf">MuJava: An Automated Class Mutation System</a> <a rel="nofollow" class="external text" href="https://web.archive.org/web/20120311010113/http://www.cs.gmu.edu/~offutt/rsrch/papers/mujava.pdf">Archived</a> 2012-03-11 at the <a href="Wayback_Machine" title="Wayback Machine">Wayback Machine</a> by Yu-Seung Ma, <a href="Jeff_Offutt" title="Jeff Offutt">Jeff Offutt</a> and Yong Rae Kwo.

</li>

<li id="cite_note-23"><a href="#cite_ref-23">^</a> <a rel="nofollow" class="external text" href="http://www.irisa.fr/manifestations/2006/Mutation2006/papers/14_Final_version.pdf">Mutation Operators for Concurrent Java (J2SE 5.0)</a> by Jeremy S. Bradbury, James R. Cordy, Juergen Dingel.

</li>

<li id="cite_note-24"><a href="#cite_ref-24">^</a> <a rel="nofollow" class="external text" href="http://www.cs.colostate.edu/~bieman/Pubs/AlexanderBiemanGhoshJiISSRE02.pdf">Mutation of Java Objects</a> by Roger T. Alexander, James M. Bieman, Sudipto Ghosh, Bixia Ji.

</li>

<li id="cite_note-25"><a href="#cite_ref-25">^</a> <a rel="nofollow" class="external text" href="http://qspace.library.queensu.ca/handle/1974/1359">Mutation-based Testing of Buffer Overflows, SQL Injections, and Format String Bugs</a> by H. Shahriar and M. Zulkernine.

</li>

<li id="cite_note-:0-26">^ <a href="#cite_ref-:0_26-0">a</a> <a href="#cite_ref-:0_26-1">b</a> <a href="#cite_ref-:0_26-2">c</a> <a href="#cite_ref-:0_26-3">d</a> <cite id="CITEREFWalters2023" class="citation web cs1">Walters, Amy (2023-06-01). <a rel="nofollow" class="external text" href="https://testrigor.com/blog/understanding-mutation-testing-a-comprehensive-guide/">"Understanding Mutation Testing: A Comprehensive Guide"</a>. testRigor AI-Based Automated Testing Tool. Retrieved 2023-10-08.</cite>

</li>

<li id="cite_note-27"><a href="#cite_ref-27">^</a> <cite id="CITEREFDengOffuttLi2013" class="citation book cs1">Deng, Lin; Offutt, Jeff; Li, Nan (22 March 2013). "Empirical Evaluation of the Statement Deletion Mutation Operator". <a rel="nofollow" class="external text" href="https://ieeexplore.ieee.org/document/6569720">2013 IEEE Sixth International Conference on Software Testing, Verification and Validation</a>. pp. 84–93. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FICST.2013.20">10.1109/ICST.2013.20</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-7695-4968-2</bdi>. <a href="ISSN_(identifier)" class="mw-redirect" title="ISSN (identifier)">ISSN</a> <a rel="nofollow" class="external text" href="https://search.worldcat.org/issn/2159-4848">2159-4848</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:12866713">12866713</a>. Retrieved 2023-10-08.</cite>

</li>

</ol></div></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hlist dl,.mw-parser-output .hlist ol,.mw-parser-output .hlist ul{margin:0;padding:0}.mw-parser-output .hlist dd,.mw-parser-output .hlist dt,.mw-parser-output .hlist li{margin:0;display:inline}.mw-parser-output .hlist.inline,.mw-parser-output .hlist.inline dl,.mw-parser-output .hlist.inline ol,.mw-parser-output .hlist.inline ul,.mw-parser-output .hlist dl dl,.mw-parser-output .hlist dl ol,.mw-parser-output .hlist dl ul,.mw-parser-output .hlist ol dl,.mw-parser-output .hlist ol ol,.mw-parser-output .hlist ol ul,.mw-parser-output .hlist ul dl,.mw-parser-output .hlist ul ol,.mw-parser-output .hlist ul ul{display:inline}.mw-parser-output .hlist .mw-empty-li{display:none}.mw-parser-output .hlist dt::after{content:": "}.mw-parser-output .hlist dd::after,.mw-parser-output .hlist li::after{content:" · ";font-weight:bold}.mw-parser-output .hlist dd:last-child::after,.mw-parser-output .hlist dt:last-child::after,.mw-parser-output .hlist li:last-child::after{content:none}.mw-parser-output .hlist dd dd:first-child::before,.mw-parser-output .hlist dd dt:first-child::before,.mw-parser-output .hlist dd li:first-child::before,.mw-parser-output .hlist dt dd:first-child::before,.mw-parser-output .hlist dt dt:first-child::before,.mw-parser-output .hlist dt li:first-child::before,.mw-parser-output .hlist li dd:first-child::before,.mw-parser-output .hlist li dt:first-child::before,.mw-parser-output .hlist li li:first-child::before{content:" (";font-weight:normal}.mw-parser-output .hlist dd dd:last-child::after,.mw-parser-output .hlist dd dt:last-child::after,.mw-parser-output .hlist dd li:last-child::after,.mw-parser-output .hlist dt dd:last-child::after,.mw-parser-output .hlist dt dt:last-child::after,.mw-parser-output .hlist dt li:last-child::after,.mw-parser-output .hlist li dd:last-child::after,.mw-parser-output .hlist li dt:last-child::after,.mw-parser-output .hlist li li:last-child::after{content:")";font-weight:normal}.mw-parser-output .hlist ol{counter-reset:listitem}.mw-parser-output .hlist ol>li{counter-increment:listitem}.mw-parser-output .hlist ol>li::before{content:" "counter(listitem)"\a0 "}.mw-parser-output .hlist dd ol>li:first-child::before,.mw-parser-output .hlist dt ol>li:first-child::before,.mw-parser-output .hlist li ol>li:first-child::before{content:" ("counter(listitem)"\a0 "}

/* end https://en.wikipedia.org/ */

</style><style data-mw-deduplicate="TemplateStyles:r1236075235">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbox{box-sizing:border-box;border:1px solid #a2a9b1;width:100%;clear:both;font-size:88%;text-align:center;padding:1px;margin:1em auto 0}.mw-parser-output .navbox .navbox{margin-top:0}.mw-parser-output .navbox+.navbox,.mw-parser-output .navbox+.navbox-styles+.navbox{margin-top:-1px}.mw-parser-output .navbox-inner,.mw-parser-output .navbox-subgroup{width:100%}.mw-parser-output .navbox-group,.mw-parser-output .navbox-title,.mw-parser-output .navbox-abovebelow{padding:0.25em 1em;line-height:1.5em;text-align:center}.mw-parser-output .navbox-group{white-space:nowrap;text-align:right}.mw-parser-output .navbox,.mw-parser-output .navbox-subgroup{background-color:#fdfdfd}.mw-parser-output .navbox-list{line-height:1.5em;border-color:#fdfdfd}.mw-parser-output .navbox-list-with-group{text-align:left;border-left-width:2px;border-left-style:solid}.mw-parser-output tr+tr>.navbox-abovebelow,.mw-parser-output tr+tr>.navbox-group,.mw-parser-output tr+tr>.navbox-image,.mw-parser-output tr+tr>.navbox-list{border-top:2px solid #fdfdfd}.mw-parser-output .navbox-title{background-color:#ccf}.mw-parser-output .navbox-abovebelow,.mw-parser-output .navbox-group,.mw-parser-output .navbox-subgroup .navbox-title{background-color:#ddf}.mw-parser-output .navbox-subgroup .navbox-group,.mw-parser-output .navbox-subgroup .navbox-abovebelow{background-color:#e6e6ff}.mw-parser-output .navbox-even{background-color:#f7f7f7}.mw-parser-output .navbox-odd{background-color:transparent}.mw-parser-output .navbox .hlist td dl,.mw-parser-output .navbox .hlist td ol,.mw-parser-output .navbox .hlist td ul,.mw-parser-output .navbox td.hlist dl,.mw-parser-output .navbox td.hlist ol,.mw-parser-output .navbox td.hlist ul{padding:0.125em 0}.mw-parser-output .navbox .navbar{display:block;font-size:100%}.mw-parser-output .navbox-title .navbar{float:left;text-align:left;margin-right:0.5em}body.skin--responsive .mw-parser-output .navbox-image img{max-width:none!important}@media print{body.ns-0 .mw-parser-output .navbox{display:none!important}}

/* end https://en.wikipedia.org/ */

</style></div><div role="navigation" class="navbox" aria-labelledby="Software_testing107" style="padding:3px"><table class="nowraplinks hlist mw-collapsible autocollapse navbox-inner" style="border-spacing:0;background:transparent;color:inherit"><tbody><tr><th scope="col" class="navbox-title" colspan="2"><style data-mw-deduplicate="TemplateStyles:r1239400231">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbar{display:inline;font-size:88%;font-weight:normal}.mw-parser-output .navbar-collapse{float:left;text-align:left}.mw-parser-output .navbar-boxtext{word-spacing:0}.mw-parser-output .navbar ul{display:inline-block;white-space:nowrap;line-height:inherit}.mw-parser-output .navbar-brackets::before{margin-right:-0.125em;content:"[ "}.mw-parser-output .navbar-brackets::after{margin-left:-0.125em;content:" ]"}.mw-parser-output .navbar li{word-spacing:-0.125em}.mw-parser-output .navbar a>span,.mw-parser-output .navbar a>abbr{text-decoration:inherit}.mw-parser-output .navbar-mini abbr{font-variant:small-caps;border-bottom:none;text-decoration:none;cursor:inherit}.mw-parser-output .navbar-ct-full{font-size:114%;margin:0 7em}.mw-parser-output .navbar-ct-mini{font-size:114%;margin:0 4em}html.skin-theme-clientpref-night .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}@media(prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}}@media print{.mw-parser-output .navbar{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div id="Software_testing107" style="font-size:114%;margin:0 4em"><a href="Software_testing" title="Software testing">Software testing</a></div></th></tr><tr><th scope="row" class="navbox-group" style="width:1%">Test levels</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Acceptance_testing" title="Acceptance testing">Acceptance testing</a></li>

<li><a href="System_integration_testing" title="System integration testing">System integration testing</a></li>

<li><a href="System_testing" title="System testing">System testing</a></li>

<li><a href="Integration_testing" title="Integration testing">Integration testing</a></li>

<li><a href="Unit_testing" title="Unit testing">Unit testing</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Test types, techniques, <a href="Software_testing_tactics" title="Software testing tactics">tactics</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><td colspan="2" class="navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="A/B_testing" title="A/B testing">A/B testing</a></li>

<li><a href="Benchmark_(computing)" title="Benchmark (computing)">Benchmark</a></li>

<li><a href="Compatibility_testing" title="Compatibility testing">Compatibility testing</a></li>

<li><a href="Concolic_testing" title="Concolic testing">Concolic testing</a></li>

<li><a href="Concurrent_testing" title="Concurrent testing">Concurrent testing</a></li>

<li><a href="Conformance_testing" title="Conformance testing">Conformance testing</a></li>

<li><a href="Continuous_testing" title="Continuous testing">Continuous testing</a></li>

<li><a href="Destructive_testing" title="Destructive testing">Destructive testing</a></li>

<li><a href="Development_testing" title="Development testing">Development testing</a></li>

<li><a href="Differential_testing" title="Differential testing">Differential testing</a></li>

<li><a href="Dynamic_program_analysis" title="Dynamic program analysis">Dynamic program analysis</a></li>

<li><a href="Installation_testing" title="Installation testing">Installation testing</a></li>

<li><a href="Negative_testing" title="Negative testing">Negative testing</a></li>

<li><a href="Random_testing" title="Random testing">Random testing</a></li>

<li><a href="Regression_testing" title="Regression testing">Regression testing</a></li>

<li><a href="Security_testing" title="Security testing">Security testing</a></li>

<li><a href="Smoke_testing_(software)" title="Smoke testing (software)">Smoke testing (software)</a></li>

<li><a href="Software_performance_testing" title="Software performance testing">Software performance testing</a></li>

<li><a href="Stress_testing_(software)" title="Stress testing (software)">Stress testing</a></li>

<li><a href="Symbolic_execution" title="Symbolic execution">Symbolic execution</a></li>

<li><a href="Test_automation" title="Test automation">Test automation</a></li>

<li><a href="Usability_testing" title="Usability testing">Usability testing</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">[x]-box style</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Black-box_testing" title="Black-box testing">Black-box testing</a>

<ul><li><a href="All-pairs_testing" title="All-pairs testing">All-pairs testing</a></li>

<li><a href="Exploratory_testing" title="Exploratory testing">Exploratory testing</a></li>

<li><a href="Fuzz_testing" class="mw-redirect" title="Fuzz testing">Fuzz testing</a></li>

<li><a href="Model-based_testing" title="Model-based testing">Model-based testing</a></li>

<li><a href="Scenario_testing" title="Scenario testing">Scenario testing</a></li></ul></li>

<li><a href="Grey-box_testing" class="mw-redirect" title="Grey-box testing">Grey-box testing</a></li>

<li><a href="White-box_testing" title="White-box testing">White-box testing</a>

<ul><li><a href="API_testing" title="API testing">API testing</a></li>

<li><a href="Static_testing" class="mw-redirect" title="Static testing">Static testing</a></li></ul></li></ul>

</div></td></tr></tbody></table><div></div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">See also</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Graphical_user_interface_testing" title="Graphical user interface testing">Graphical user interface testing</a></li>

<li><a href="Manual_testing" title="Manual testing">Manual testing</a></li>

<li><a href="Orthogonal_array_testing" title="Orthogonal array testing">Orthogonal array testing</a></li>

<li><a href="Pair_testing" title="Pair testing">Pair testing</a></li>

<li><a href="Soak_testing" title="Soak testing">Soak testing</a></li>

<li><a href="Software_reliability_testing" title="Software reliability testing">Software reliability testing</a></li>

<li><a href="Stress_testing" title="Stress testing">Stress testing</a></li>

<li><a href="Web_testing" title="Web testing">Web testing</a></li></ul>

</div></td></tr></tbody></table></div></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2025-06-04" href="https://en.wikipedia.org/wiki/?title=Mutation_testing&oldid=1293967810">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>